{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "metadata": false
    }
   },
   "source": [
    "# FL client Joining FL experiment \n",
    "\n",
    "The purpose of this notebook is to show how to start clients to participate in an FL experiment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "metadata": false
    }
   },
   "source": [
    "## Prerequisites\n",
    "- The [Startup notebook](1-Startup.ipynb) has been run successfully.\n",
    "- A server has been started."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Edit Docker Script\n",
    "\n",
    "Before starting the docker script, you need to edit it to ensure that the environments (such as dataset, GPUs, memory, ...) meet your requirement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from IPython.display import HTML\n",
    "from multiprocessing import Process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "client_name_1 = \"org1-a\"\n",
    "client_name_2 = \"org1-b\"\n",
    "workspace = \"demo_workspace\"\n",
    "\n",
    "client1_startup_path = os.path.join(workspace, client_name_1, 'startup')\n",
    "client2_startup_path = os.path.join(workspace, client_name_2, 'startup')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Please click here to check the org1-a script](demo_workspace/org1-a/startup/docker.sh)\n",
    "\n",
    "[Please click here to check the org1-b script](demo_workspace/org1-b/startup/docker.sh)\n",
    "\n",
    "As we can see, the default data directory is `MY_DATA_DIR=/home/flclient/data/msd-data/Task09_Spleen`.\n",
    "\n",
    "Please modify it and ensure that it equals to the actual path that contains `Task09_Spleen`. If you downloaded the dataset via using `prerpare_expr_files.sh`, `MY_DATA_DIR` can be set to `$(pwd)`. In addition, you may also need to change `GPU2USE` and add `shm-size` argument for `docker run` command.\n",
    "\n",
    "The following is my modified script for `org1-a`:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "#!/usr/bin/env bash\n",
    "DIR=\"$( cd \"$( dirname \"${BASH_SOURCE[0]}\" )\" >/dev/null 2>&1 && pwd )\"\n",
    "\n",
    "# docker run script for FL client\n",
    "# local data directory\n",
    "MY_DATA_DIR=$(pwd)\n",
    "# for all gpus use line below \n",
    "#GPU2USE=all \n",
    "# for 2 gpus use line below\n",
    "#GPU2USE=2 \n",
    "# for specific gpus as gpu#0 and gpu#2 use line below\n",
    "GPU2USE='\"device=0\"'\n",
    "# to use host network, use line below\n",
    "NETARG=\"--net=host\"\n",
    "# FL clients do not need to open ports, so the following line is not needed.\n",
    "#NETARG=\"-p 443:443 -p 8003:8003\"\n",
    "DOCKER_IMAGE=monai_nvflare:latest\n",
    "echo \"Starting docker with $DOCKER_IMAGE\"\n",
    "docker run --rm -it --shm-size 16G --name=org1-a --gpus=$GPU2USE -u $(id -u):$(id -g) -v /etc/passwd:/etc/passwd -v /etc/group:/etc/group -v $DIR/..:/workspace/ -v $MY_DATA_DIR:/data/:ro -w /workspace/ --ipc=host $NETARG $DOCKER_IMAGE /bin/bash\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check the Connection to Server\n",
    "\n",
    "Run telnet with the server port. A successful connection should give a message as shown below.\n",
    "\n",
    "```\n",
    "Trying 127.0.0.1...\n",
    "Connected to localhost.\n",
    "Escape character is '^]'.\n",
    "Connection closed by foreign host.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "pycharm": {
     "metadata": false,
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trying 127.0.0.1...\n",
      "Connected to localhost.\n",
      "Escape character is '^]'.\n",
      "^C\n",
      "Connection closed by foreign host.\n"
     ]
    }
   ],
   "source": [
    "!telnet localhost 8002"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "metadata": false
    }
   },
   "source": [
    "### Start Clients Docker\n",
    "\n",
    "After modifying `docker.sh` for `org1-a` and `org1-b`, we are able to start:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "pycharm": {
     "metadata": false,
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "running cmd demo_workspace/org1-b/startup/docker.sh\n",
      "running cmd demo_workspace/org1-a/startup/docker.sh\n",
      "Starting docker with monai_nvflare:latest\n",
      "Starting docker with monai_nvflare:latest\n",
      "\n",
      "=============\n",
      "== PyTorch ==\n",
      "=============\n",
      "\n",
      "NVIDIA Release 21.08 (build 26011915)\n",
      "PyTorch Version 1.10.0a0+3fd9dcf\n",
      "\n",
      "Container image Copyright (c) 2021, NVIDIA CORPORATION.  All rights reserved.\n",
      "\n",
      "Copyright (c) 2014-2021 Facebook Inc.\n",
      "Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)\n",
      "Copyright (c) 2012-2014 Deepmind Technologies    (Koray Kavukcuoglu)\n",
      "Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)\n",
      "Copyright (c) 2011-2013 NYU                      (Clement Farabet)\n",
      "Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)\n",
      "Copyright (c) 2006      Idiap Research Institute (Samy Bengio)\n",
      "Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)\n",
      "Copyright (c) 2015      Google Inc.\n",
      "Copyright (c) 2015      Yangqing Jia\n",
      "Copyright (c) 2013-2016 The Caffe contributors\n",
      "All rights reserved.\n",
      "\n",
      "NVIDIA Deep Learning Profiler (dlprof) Copyright (c) 2021, NVIDIA CORPORATION.  All rights reserved.\n",
      "\n",
      "Various files include modifications (c) NVIDIA CORPORATION.  All rights reserved.\n",
      "\n",
      "This container image and its contents are governed by the NVIDIA Deep Learning Container License.\n",
      "By pulling and using the container, you accept the terms and conditions of this license:\n",
      "https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license\n",
      "\n",
      "=============\n",
      "== PyTorch ==\n",
      "=============\n",
      "\n",
      "NVIDIA Release 21.08 (build 26011915)\n",
      "PyTorch Version 1.10.0a0+3fd9dcf\n",
      "\n",
      "Container image Copyright (c) 2021, NVIDIA CORPORATION.  All rights reserved.\n",
      "\n",
      "Copyright (c) 2014-2021 Facebook Inc.\n",
      "Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)\n",
      "Copyright (c) 2012-2014 Deepmind Technologies    (Koray Kavukcuoglu)\n",
      "Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)\n",
      "Copyright (c) 2011-2013 NYU                      (Clement Farabet)\n",
      "Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)\n",
      "Copyright (c) 2006      Idiap Research Institute (Samy Bengio)\n",
      "Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)\n",
      "Copyright (c) 2015      Google Inc.\n",
      "Copyright (c) 2015      Yangqing Jia\n",
      "Copyright (c) 2013-2016 The Caffe contributors\n",
      "All rights reserved.\n",
      "\n",
      "NVIDIA Deep Learning Profiler (dlprof) Copyright (c) 2021, NVIDIA CORPORATION.  All rights reserved.\n",
      "\n",
      "Various files include modifications (c) NVIDIA CORPORATION.  All rights reserved.\n",
      "\n",
      "This container image and its contents are governed by the NVIDIA Deep Learning Container License.\n",
      "By pulling and using the container, you accept the terms and conditions of this license:\n",
      "https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license\n",
      "\n",
      "NOTE: MOFED driver for multi-node communication was not detected.\n",
      "      Multi-node communication performance may be reduced.\n",
      "\n",
      "venn@sys:/workspace$ \n",
      "NOTE: MOFED driver for multi-node communication was not detected.\n",
      "      Multi-node communication performance may be reduced.\n",
      "\n",
      "venn@sys:/workspace$ "
     ]
    }
   ],
   "source": [
    "def run_client1():\n",
    "    cmd = client1_startup_path + \"/docker.sh\"\n",
    "    print(\"running cmd \" + cmd)\n",
    "    !$cmd\n",
    "\n",
    "\n",
    "def run_client2():\n",
    "    cmd = client2_startup_path + \"/docker.sh\"\n",
    "    print(\"running cmd \" + cmd)\n",
    "    !$cmd\n",
    "\n",
    "\n",
    "p1 = Process(target=run_client1)\n",
    "p2 = Process(target=run_client2)\n",
    "\n",
    "p1.start()\n",
    "p2.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check Started Containers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CONTAINER ID   IMAGE                  COMMAND                  CREATED          STATUS          PORTS     NAMES\n",
      "96c8f9406303   monai_nvflare:latest   \"/usr/local/bin/nvid…\"   2 seconds ago    Up 1 second               org1-a\n",
      "6b5d9eefc699   monai_nvflare:latest   \"/usr/local/bin/nvid…\"   2 seconds ago    Up 1 second               org1-b\n",
      "08492335d003   monai_nvflare:latest   \"/usr/local/bin/nvid…\"   40 seconds ago   Up 39 seconds             flserver\n"
     ]
    }
   ],
   "source": [
    "!docker ps -a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Start Clients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To start a client, you should:\n",
    "\n",
    "- open a terminal and enter the container named `org1-a`.\n",
    "- run `start.sh` under `startup/`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<a href=\"\", data-commandlinker-command=\"terminal:create-new\"> Open a new terminal</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# You can click the following link, or manually open a new terminal.\n",
    "HTML('<a href=\"\", data-commandlinker-command=\"terminal:create-new\"> Open a new terminal</a>')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "docker exec -it org1-a bash\n",
    "cd startup/\n",
    "sh start.shThe commands can be:\n",
    "\n",
    "```\n",
    "docker exec -it org1-a bash\n",
    "cd startup/\n",
    "sh start.sh\n",
    "```\n",
    "\n",
    "A successfully started client will print logs as follow:\n",
    "<br>![fl](demo_figs/enter_client_success.png)<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To start the second client, please open a new terminal again, and just use the same commands but for the `org1-b` Docker container:\n",
    "```\n",
    "docker exec -it org1-b bash\n",
    "cd startup/\n",
    "sh start.sh\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After a client has been successfully started, the server side will show the following information:\n",
    "<br>![fl](demo_figs/successful_regist_clients.png)<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Next Steps\n",
    "\n",
    "You have now started two client containers.\n",
    "In the next notebook, [Admin Startup Notebook](4-Admin.ipynb), you'll start an admin participating in the FL experiment."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "stem_cell": {
   "cell_type": "raw",
   "metadata": {
    "pycharm": {
     "metadata": false
    }
   },
   "source": ""
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
